{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../../docs/images/DSPy8.png\" alt=\"DSPy7 Image\" height=\"120\"/>\n",
    "\n",
    "### Multi-Agent DSPy Programs: Bootstrapping & Aggregating Multiple `ReAct` Agents\n",
    "\n",
    "This is a quick (somewhat advanced) example of DSPy. You're given a hard QA task and an agent architecture (`dspy.ReAct`), how do you get high scores without tinkering with prompts?\n",
    "\n",
    "There are many ways, but this notebook shows one complex strategy that DSPy makes near-trivial to achieve: we'll automatically bootstrap five different highly-effective prompts for ReAct, then optimize an aggregator that combines their powers.\n",
    "\n",
    "As is usually the case with DSPy, the code to do this is probably shorter than describing it in English, so let's jump right into that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0) TLDR.\n",
    "\n",
    "This is a copy of the `multi_agent.ipynb` example in the same directory, but now with **Llama3-8b** instead of **GPT-3.5**.\n",
    "\n",
    "We'll build a ReAct agent in DSPy that scores 24% accuracy on a retrieval-based question answering task.\n",
    "\n",
    "Then, we'll optimize it with `BootstrapFewShotWithRandomSearch` to get 35% accuracy.\n",
    "\n",
    "Then, we'll build a multi-agent aggregator over five different optimized versions of the agent.\n",
    "\n",
    "Our unoptimized aggregator will score 21%. It doesn't understand the task. Hence, we'll optimize the aggregator too.\n",
    "\n",
    "We'll end up with an optimized multi-agent system that scores a whopping 59% accuracy on the same task.\n",
    "\n",
    "The core portion of the code to do this can be fit into 15 lines of DSPy, but we'll sprinkle some short explanations below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Setting Up.\n",
    "\n",
    "We'll configure the language model (GPT-3.5) and the retrieval model (ColBERTv2 over Wikipedia)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dspy\n",
    "from dspy.evaluate import Evaluate\n",
    "from dspy.datasets.hotpotqa import HotPotQA\n",
    "from dspy.teleprompt import BootstrapFewShotWithRandomSearch\n",
    "\n",
    "# Set up llama3 with a VLLM client, served on four GPUs. Please note that these URLs will not work for you; you'd need to refer to the documentation to set up your own VLLM/SGLANG server(s).\n",
    "llama3 = dspy.HFClientVLLM(model=\"meta-llama/Meta-Llama-3-8B-Instruct\", port=None, url=[\"http://future-hgx-3:7411\", \"http://future-hgx-3:7412\", \"http://future-hgx-3:7413\", \"http://future-hgx-1:7414\"], max_tokens=500, stop=('\\n',))\n",
    "colbert = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')\n",
    "dspy.configure(lm=llama3, rm=colbert)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Loading some data.\n",
    "\n",
    "We'll load 150 examples for training (`trainset`), 50 examples for validation & optimization (`valset`), and 300 examples for evaluation (`devset`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Example({'question': 'At My Window was released by which American singer-songwriter?', 'answer': 'John Townes Van Zandt'}) (input_keys={'question'})"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = HotPotQA(train_seed=1, train_size=200, eval_seed=2023, dev_size=300, test_size=0)\n",
    "trainset = [x.with_inputs('question') for x in dataset.train[0:150]]\n",
    "valset = [x.with_inputs('question') for x in dataset.train[150:200]]\n",
    "devset = [x.with_inputs('question') for x in dataset.dev]\n",
    "\n",
    "# show an example datapoint; it's just a question-answer pair\n",
    "trainset[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) ReAct Agent.\n",
    "\n",
    "Our agent will just be a DSPy ReAct agent that takes a `question` and outputs the `answer` by using a ColBERTv2 retrieval tool."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = dspy.ReAct(\"question -> answer\", tools=[dspy.Retrieve(k=1)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's evaluate this **unoptimized** ReAct agent on the `devset`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/300 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 72 / 300  (24.0): 100%|██████████| 300/300 [01:37<00:00,  3.07it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_fc33c th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_fc33c td {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_fc33c_row0_col0, #T_fc33c_row0_col1, #T_fc33c_row0_col2, #T_fc33c_row0_col3, #T_fc33c_row0_col4, #T_fc33c_row0_col5, #T_fc33c_row1_col0, #T_fc33c_row1_col1, #T_fc33c_row1_col2, #T_fc33c_row1_col3, #T_fc33c_row1_col4, #T_fc33c_row1_col5, #T_fc33c_row2_col0, #T_fc33c_row2_col1, #T_fc33c_row2_col2, #T_fc33c_row2_col3, #T_fc33c_row2_col4, #T_fc33c_row2_col5, #T_fc33c_row3_col0, #T_fc33c_row3_col1, #T_fc33c_row3_col2, #T_fc33c_row3_col3, #T_fc33c_row3_col4, #T_fc33c_row3_col5, #T_fc33c_row4_col0, #T_fc33c_row4_col1, #T_fc33c_row4_col2, #T_fc33c_row4_col3, #T_fc33c_row4_col4, #T_fc33c_row4_col5 {\n",
       "  text-align: left;\n",
       "  white-space: pre-wrap;\n",
       "  word-wrap: break-word;\n",
       "  max-width: 400px;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_fc33c\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_fc33c_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
       "      <th id=\"T_fc33c_level0_col1\" class=\"col_heading level0 col1\" >example_answer</th>\n",
       "      <th id=\"T_fc33c_level0_col2\" class=\"col_heading level0 col2\" >gold_titles</th>\n",
       "      <th id=\"T_fc33c_level0_col3\" class=\"col_heading level0 col3\" >observations</th>\n",
       "      <th id=\"T_fc33c_level0_col4\" class=\"col_heading level0 col4\" >pred_answer</th>\n",
       "      <th id=\"T_fc33c_level0_col5\" class=\"col_heading level0 col5\" >answer_exact_match</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_fc33c_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_fc33c_row0_col0\" class=\"data row0 col0\" >Are both Cangzhou and Qionghai in the Hebei province of China?</td>\n",
       "      <td id=\"T_fc33c_row0_col1\" class=\"data row0 col1\" >no</td>\n",
       "      <td id=\"T_fc33c_row0_col2\" class=\"data row0 col2\" >{'Qionghai', 'Cangzhou'}</td>\n",
       "      <td id=\"T_fc33c_row0_col3\" class=\"data row0 col3\" >[['Hebei | Hebei (; postal: Hopeh) is a province of China in the North China region. Its one-character abbreviation is \"冀 \" (Jì), named after...</td>\n",
       "      <td id=\"T_fc33c_row0_col4\" class=\"data row0 col4\" >No, Qionghai is not in the Hebei province of China</td>\n",
       "      <td id=\"T_fc33c_row0_col5\" class=\"data row0 col5\" >False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_fc33c_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_fc33c_row1_col0\" class=\"data row1 col0\" >Who conducts the draft in which Marc-Andre Fleury was drafted to the Vegas Golden Knights for the 2017-18 season?</td>\n",
       "      <td id=\"T_fc33c_row1_col1\" class=\"data row1 col1\" >National Hockey League</td>\n",
       "      <td id=\"T_fc33c_row1_col2\" class=\"data row1 col2\" >{'2017–18 Pittsburgh Penguins season', '2017 NHL Expansion Draft'}</td>\n",
       "      <td id=\"T_fc33c_row1_col3\" class=\"data row1 col3\" >[['2017 NHL Entry Draft | The 2017 NHL Entry Draft was the 55th NHL Entry Draft. The draft was held from June 23–24, 2017, at...</td>\n",
       "      <td id=\"T_fc33c_row1_col4\" class=\"data row1 col4\" >answer=\"Vegas Golden Knights\"</td>\n",
       "      <td id=\"T_fc33c_row1_col5\" class=\"data row1 col5\" >False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_fc33c_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_fc33c_row2_col0\" class=\"data row2 col0\" >The Wings entered a new era, following the retirement of which Canadian retired professional ice hockey player and current general manager of the Tampa Bay...</td>\n",
       "      <td id=\"T_fc33c_row2_col1\" class=\"data row2 col1\" >Steve Yzerman</td>\n",
       "      <td id=\"T_fc33c_row2_col2\" class=\"data row2 col2\" >{'2006–07 Detroit Red Wings season', 'Steve Yzerman'}</td>\n",
       "      <td id=\"T_fc33c_row2_col3\" class=\"data row2 col3\" >[['Jay Feaster | Jay Harry Feaster (born July 30, 1962 in Harrisburg, Pennsylvania) is a National Hockey League (NHL) executive currently serving as the Executive...</td>\n",
       "      <td id=\"T_fc33c_row2_col4\" class=\"data row2 col4\" >Steve Yzerman</td>\n",
       "      <td id=\"T_fc33c_row2_col5\" class=\"data row2 col5\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_fc33c_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_fc33c_row3_col0\" class=\"data row3 col0\" >What river is near the Crichton Collegiate Church?</td>\n",
       "      <td id=\"T_fc33c_row3_col1\" class=\"data row3 col1\" >the River Tyne</td>\n",
       "      <td id=\"T_fc33c_row3_col2\" class=\"data row3 col2\" >{'Crichton Collegiate Church', 'Crichton Castle'}</td>\n",
       "      <td id=\"T_fc33c_row3_col3\" class=\"data row3 col3\" >[[\"Crichton Collegiate Church | Crichton Collegiate Church is situated about 0.6 mi south west of the hamlet of Crichton in Midlothian, Scotland. Crichton itself is...</td>\n",
       "      <td id=\"T_fc33c_row3_col4\" class=\"data row3 col4\" >River Tyne</td>\n",
       "      <td id=\"T_fc33c_row3_col5\" class=\"data row3 col5\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_fc33c_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_fc33c_row4_col0\" class=\"data row4 col0\" >In the 10th Century A.D. Ealhswith had a son called Æthelweard by which English king?</td>\n",
       "      <td id=\"T_fc33c_row4_col1\" class=\"data row4 col1\" >King Alfred the Great</td>\n",
       "      <td id=\"T_fc33c_row4_col2\" class=\"data row4 col2\" >{'Æthelweard (son of Alfred)', 'Ealhswith'}</td>\n",
       "      <td id=\"T_fc33c_row4_col3\" class=\"data row4 col3\" >[['Æthelweard (son of Alfred) | Æthelweard (d. 920 or 922) was the younger son of King Alfred the Great and Ealhswith.'], ['Ealhswith | Ealhswith or...</td>\n",
       "      <td id=\"T_fc33c_row4_col4\" class=\"data row4 col4\" >King Alfred the Great</td>\n",
       "      <td id=\"T_fc33c_row4_col5\" class=\"data row4 col5\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7fa584c9d9f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <div style='\n",
       "                    text-align: center;\n",
       "                    font-size: 16px;\n",
       "                    font-weight: bold;\n",
       "                    color: #555;\n",
       "                    margin: 10px 0;'>\n",
       "                    ... 295 more rows not displayed ...\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "24.0"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set up an evaluator on the first 300 examples of the devset.\n",
    "config = dict(num_threads=8, display_progress=True, display_table=5)\n",
    "evaluate = Evaluate(devset=devset, metric=dspy.evaluate.answer_exact_match, **config)\n",
    "\n",
    "evaluate(agent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4) Optimized ReAct.\n",
    "\n",
    "Let's use DSPy's simple `BootstrapFewShotWithRandomSearch` optimizer to create successful examples of the ReAct program and attempt to optimize the prompts using those constructed examples. In the future, we could try more sophisticated DSPy optimizers too, like `MIPRO`.\n",
    "\n",
    "We'll bootstrap 20 programs that way. Examples will be bootstrapped starting from the `trainset` and optimized over our tiny `valset`. We'll evaluate later on the `devset`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 2 / 3  (66.7):   4%|▍         | 2/50 [00:00<00:00, 77.81it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 15 / 50  (30.0): 100%|██████████| 50/50 [00:09<00:00,  5.29it/s]\n",
      "Average Metric: 15 / 50  (30.0): 100%|██████████| 50/50 [00:06<00:00,  7.63it/s]\n",
      "  5%|▌         | 8/150 [00:26<07:55,  3.35s/it]\n",
      "Average Metric: 19 / 50  (38.0): 100%|██████████| 50/50 [00:16<00:00,  3.10it/s]\n",
      "  5%|▌         | 8/150 [00:25<07:33,  3.19s/it]\n",
      "Average Metric: 16 / 50  (32.0): 100%|██████████| 50/50 [00:20<00:00,  2.47it/s]\n",
      "  2%|▏         | 3/150 [00:08<07:20,  2.99s/it]\n",
      "Average Metric: 16 / 50  (32.0): 100%|██████████| 50/50 [00:17<00:00,  2.92it/s]\n",
      "  1%|▏         | 2/150 [00:02<02:36,  1.06s/it]\n",
      "Average Metric: 16 / 50  (32.0): 100%|██████████| 50/50 [00:09<00:00,  5.36it/s]\n",
      "  5%|▍         | 7/150 [00:22<07:32,  3.16s/it]\n",
      "Average Metric: 16 / 50  (32.0): 100%|██████████| 50/50 [00:07<00:00,  6.33it/s]\n",
      "  8%|▊         | 12/150 [00:37<07:12,  3.13s/it]\n",
      "Average Metric: 18 / 50  (36.0): 100%|██████████| 50/50 [00:14<00:00,  3.39it/s]\n"
     ]
    }
   ],
   "source": [
    "config = dict(max_bootstrapped_demos=2, max_labeled_demos=0, num_candidate_programs=5, num_threads=8)\n",
    "tp = BootstrapFewShotWithRandomSearch(metric=dspy.evaluate.answer_exact_match, **config)\n",
    "optimized_react = tp.compile(agent, trainset=trainset, valset=valset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/300 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 93 / 300  (31.0): 100%|██████████| 300/300 [02:08<00:00,  2.33it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_dc573 th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_dc573 td {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_dc573_row0_col0, #T_dc573_row0_col1, #T_dc573_row0_col2, #T_dc573_row0_col3, #T_dc573_row0_col4, #T_dc573_row0_col5, #T_dc573_row1_col0, #T_dc573_row1_col1, #T_dc573_row1_col2, #T_dc573_row1_col3, #T_dc573_row1_col4, #T_dc573_row1_col5, #T_dc573_row2_col0, #T_dc573_row2_col1, #T_dc573_row2_col2, #T_dc573_row2_col3, #T_dc573_row2_col4, #T_dc573_row2_col5, #T_dc573_row3_col0, #T_dc573_row3_col1, #T_dc573_row3_col2, #T_dc573_row3_col3, #T_dc573_row3_col4, #T_dc573_row3_col5, #T_dc573_row4_col0, #T_dc573_row4_col1, #T_dc573_row4_col2, #T_dc573_row4_col3, #T_dc573_row4_col4, #T_dc573_row4_col5 {\n",
       "  text-align: left;\n",
       "  white-space: pre-wrap;\n",
       "  word-wrap: break-word;\n",
       "  max-width: 400px;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_dc573\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_dc573_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
       "      <th id=\"T_dc573_level0_col1\" class=\"col_heading level0 col1\" >example_answer</th>\n",
       "      <th id=\"T_dc573_level0_col2\" class=\"col_heading level0 col2\" >gold_titles</th>\n",
       "      <th id=\"T_dc573_level0_col3\" class=\"col_heading level0 col3\" >observations</th>\n",
       "      <th id=\"T_dc573_level0_col4\" class=\"col_heading level0 col4\" >pred_answer</th>\n",
       "      <th id=\"T_dc573_level0_col5\" class=\"col_heading level0 col5\" >answer_exact_match</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_dc573_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_dc573_row0_col0\" class=\"data row0 col0\" >Are both Cangzhou and Qionghai in the Hebei province of China?</td>\n",
       "      <td id=\"T_dc573_row0_col1\" class=\"data row0 col1\" >no</td>\n",
       "      <td id=\"T_dc573_row0_col2\" class=\"data row0 col2\" >{'Qionghai', 'Cangzhou'}</td>\n",
       "      <td id=\"T_dc573_row0_col3\" class=\"data row0 col3\" >[]</td>\n",
       "      <td id=\"T_dc573_row0_col4\" class=\"data row0 col4\" >no</td>\n",
       "      <td id=\"T_dc573_row0_col5\" class=\"data row0 col5\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_dc573_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_dc573_row1_col0\" class=\"data row1 col0\" >Who conducts the draft in which Marc-Andre Fleury was drafted to the Vegas Golden Knights for the 2017-18 season?</td>\n",
       "      <td id=\"T_dc573_row1_col1\" class=\"data row1 col1\" >National Hockey League</td>\n",
       "      <td id=\"T_dc573_row1_col2\" class=\"data row1 col2\" >{'2017–18 Pittsburgh Penguins season', '2017 NHL Expansion Draft'}</td>\n",
       "      <td id=\"T_dc573_row1_col3\" class=\"data row1 col3\" >[[\"2017 NHL Expansion Draft | The 2017 NHL Expansion Draft was an expansion draft conducted by the National Hockey League on June 18–20, 2017 to...</td>\n",
       "      <td id=\"T_dc573_row1_col4\" class=\"data row1 col4\" >George McPhee</td>\n",
       "      <td id=\"T_dc573_row1_col5\" class=\"data row1 col5\" >False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_dc573_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_dc573_row2_col0\" class=\"data row2 col0\" >The Wings entered a new era, following the retirement of which Canadian retired professional ice hockey player and current general manager of the Tampa Bay...</td>\n",
       "      <td id=\"T_dc573_row2_col1\" class=\"data row2 col1\" >Steve Yzerman</td>\n",
       "      <td id=\"T_dc573_row2_col2\" class=\"data row2 col2\" >{'2006–07 Detroit Red Wings season', 'Steve Yzerman'}</td>\n",
       "      <td id=\"T_dc573_row2_col3\" class=\"data row2 col3\" >[['Gretzky (disambiguation) | Wayne Gretzky is a retired National Hockey League player.'], [\"2006–07 Detroit Red Wings season | The 2006–07 Detroit Red Wings season was...</td>\n",
       "      <td id=\"T_dc573_row2_col4\" class=\"data row2 col4\" >\"Steve Yzerman\"</td>\n",
       "      <td id=\"T_dc573_row2_col5\" class=\"data row2 col5\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_dc573_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_dc573_row3_col0\" class=\"data row3 col0\" >What river is near the Crichton Collegiate Church?</td>\n",
       "      <td id=\"T_dc573_row3_col1\" class=\"data row3 col1\" >the River Tyne</td>\n",
       "      <td id=\"T_dc573_row3_col2\" class=\"data row3 col2\" >{'Crichton Collegiate Church', 'Crichton Castle'}</td>\n",
       "      <td id=\"T_dc573_row3_col3\" class=\"data row3 col3\" >[['Dumfries | Dumfries ( ; possibly from Scottish Gaelic: \"Dùn Phris\" ) is a market town and former royal burgh within the Dumfries and Galloway...</td>\n",
       "      <td id=\"T_dc573_row3_col4\" class=\"data row3 col4\" >River Nith</td>\n",
       "      <td id=\"T_dc573_row3_col5\" class=\"data row3 col5\" >False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_dc573_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_dc573_row4_col0\" class=\"data row4 col0\" >In the 10th Century A.D. Ealhswith had a son called Æthelweard by which English king?</td>\n",
       "      <td id=\"T_dc573_row4_col1\" class=\"data row4 col1\" >King Alfred the Great</td>\n",
       "      <td id=\"T_dc573_row4_col2\" class=\"data row4 col2\" >{'Æthelweard (son of Alfred)', 'Ealhswith'}</td>\n",
       "      <td id=\"T_dc573_row4_col3\" class=\"data row4 col3\" >[['Ealhswith | Ealhswith or Ealswitha (died 5 December 902) was the wife of King Alfred the Great. Her father was a Mercian nobleman, Æthelred Mucel,...</td>\n",
       "      <td id=\"T_dc573_row4_col4\" class=\"data row4 col4\" >\"Edward the Elder\"</td>\n",
       "      <td id=\"T_dc573_row4_col5\" class=\"data row4 col5\" >False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7fa5a54a23b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <div style='\n",
       "                    text-align: center;\n",
       "                    font-size: 16px;\n",
       "                    font-weight: bold;\n",
       "                    color: #555;\n",
       "                    margin: 10px 0;'>\n",
       "                    ... 295 more rows not displayed ...\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "31.0"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(optimized_react)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimized_reactX = optimized_react.deepcopy()\n",
    "del optimized_reactX.candidate_programs\n",
    "\n",
    "config = dict(max_bootstrapped_demos=2, max_labeled_demos=0, num_candidate_programs=20, num_threads=8)\n",
    "tp = BootstrapFewShotWithRandomSearch(metric=dspy.evaluate.answer_exact_match, **config)\n",
    "optimized_react2 = tp.compile(agent, trainset=trainset, valset=valset, teacher=optimized_reactX)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 105 / 300  (35.0): 100%|██████████| 300/300 [01:10<00:00,  4.27it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_babe2 th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_babe2 td {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_babe2_row0_col0, #T_babe2_row0_col1, #T_babe2_row0_col2, #T_babe2_row0_col3, #T_babe2_row0_col4, #T_babe2_row0_col5, #T_babe2_row1_col0, #T_babe2_row1_col1, #T_babe2_row1_col2, #T_babe2_row1_col3, #T_babe2_row1_col4, #T_babe2_row1_col5, #T_babe2_row2_col0, #T_babe2_row2_col1, #T_babe2_row2_col2, #T_babe2_row2_col3, #T_babe2_row2_col4, #T_babe2_row2_col5, #T_babe2_row3_col0, #T_babe2_row3_col1, #T_babe2_row3_col2, #T_babe2_row3_col3, #T_babe2_row3_col4, #T_babe2_row3_col5, #T_babe2_row4_col0, #T_babe2_row4_col1, #T_babe2_row4_col2, #T_babe2_row4_col3, #T_babe2_row4_col4, #T_babe2_row4_col5 {\n",
       "  text-align: left;\n",
       "  white-space: pre-wrap;\n",
       "  word-wrap: break-word;\n",
       "  max-width: 400px;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_babe2\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_babe2_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
       "      <th id=\"T_babe2_level0_col1\" class=\"col_heading level0 col1\" >example_answer</th>\n",
       "      <th id=\"T_babe2_level0_col2\" class=\"col_heading level0 col2\" >gold_titles</th>\n",
       "      <th id=\"T_babe2_level0_col3\" class=\"col_heading level0 col3\" >observations</th>\n",
       "      <th id=\"T_babe2_level0_col4\" class=\"col_heading level0 col4\" >pred_answer</th>\n",
       "      <th id=\"T_babe2_level0_col5\" class=\"col_heading level0 col5\" >answer_exact_match</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_babe2_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_babe2_row0_col0\" class=\"data row0 col0\" >Are both Cangzhou and Qionghai in the Hebei province of China?</td>\n",
       "      <td id=\"T_babe2_row0_col1\" class=\"data row0 col1\" >no</td>\n",
       "      <td id=\"T_babe2_row0_col2\" class=\"data row0 col2\" >{'Qionghai', 'Cangzhou'}</td>\n",
       "      <td id=\"T_babe2_row0_col3\" class=\"data row0 col3\" >[['Cangzhou | Cangzhou () is a prefecture-level city in eastern Hebei province, People\\'s Republic of China. At the 2010 census, Cangzhou\\'s built-up (\"or metro\") area...</td>\n",
       "      <td id=\"T_babe2_row0_col4\" class=\"data row0 col4\" >no</td>\n",
       "      <td id=\"T_babe2_row0_col5\" class=\"data row0 col5\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_babe2_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_babe2_row1_col0\" class=\"data row1 col0\" >Who conducts the draft in which Marc-Andre Fleury was drafted to the Vegas Golden Knights for the 2017-18 season?</td>\n",
       "      <td id=\"T_babe2_row1_col1\" class=\"data row1 col1\" >National Hockey League</td>\n",
       "      <td id=\"T_babe2_row1_col2\" class=\"data row1 col2\" >{'2017–18 Pittsburgh Penguins season', '2017 NHL Expansion Draft'}</td>\n",
       "      <td id=\"T_babe2_row1_col3\" class=\"data row1 col3\" >[[\"2017 NHL Expansion Draft | The 2017 NHL Expansion Draft was an expansion draft conducted by the National Hockey League on June 18–20, 2017 to...</td>\n",
       "      <td id=\"T_babe2_row1_col4\" class=\"data row1 col4\" >George McPhee</td>\n",
       "      <td id=\"T_babe2_row1_col5\" class=\"data row1 col5\" >False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_babe2_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_babe2_row2_col0\" class=\"data row2 col0\" >The Wings entered a new era, following the retirement of which Canadian retired professional ice hockey player and current general manager of the Tampa Bay...</td>\n",
       "      <td id=\"T_babe2_row2_col1\" class=\"data row2 col1\" >Steve Yzerman</td>\n",
       "      <td id=\"T_babe2_row2_col2\" class=\"data row2 col2\" >{'2006–07 Detroit Red Wings season', 'Steve Yzerman'}</td>\n",
       "      <td id=\"T_babe2_row2_col3\" class=\"data row2 col3\" >[['Tony Resch | Tony Resch is a retired lacrosse player, and current field and box lacrosse head coach. He is the former head coach of...</td>\n",
       "      <td id=\"T_babe2_row2_col4\" class=\"data row2 col4\" >Steve Yzerman</td>\n",
       "      <td id=\"T_babe2_row2_col5\" class=\"data row2 col5\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_babe2_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_babe2_row3_col0\" class=\"data row3 col0\" >What river is near the Crichton Collegiate Church?</td>\n",
       "      <td id=\"T_babe2_row3_col1\" class=\"data row3 col1\" >the River Tyne</td>\n",
       "      <td id=\"T_babe2_row3_col2\" class=\"data row3 col2\" >{'Crichton Collegiate Church', 'Crichton Castle'}</td>\n",
       "      <td id=\"T_babe2_row3_col3\" class=\"data row3 col3\" >[[\"Crichton Collegiate Church | Crichton Collegiate Church is situated about 0.6 mi south west of the hamlet of Crichton in Midlothian, Scotland. Crichton itself is...</td>\n",
       "      <td id=\"T_babe2_row3_col4\" class=\"data row3 col4\" >River Tyne</td>\n",
       "      <td id=\"T_babe2_row3_col5\" class=\"data row3 col5\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_babe2_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_babe2_row4_col0\" class=\"data row4 col0\" >In the 10th Century A.D. Ealhswith had a son called Æthelweard by which English king?</td>\n",
       "      <td id=\"T_babe2_row4_col1\" class=\"data row4 col1\" >King Alfred the Great</td>\n",
       "      <td id=\"T_babe2_row4_col2\" class=\"data row4 col2\" >{'Æthelweard (son of Alfred)', 'Ealhswith'}</td>\n",
       "      <td id=\"T_babe2_row4_col3\" class=\"data row4 col3\" >[['Steve Ellsworth | Steven Clark Ellsworth (born July 30, 1960 in Chicago) is the son of Dick Ellsworth and is a former Major League Baseball...</td>\n",
       "      <td id=\"T_babe2_row4_col4\" class=\"data row4 col4\" >answer</td>\n",
       "      <td id=\"T_babe2_row4_col5\" class=\"data row4 col5\" >False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7fa56d7e07c0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <div style='\n",
       "                    text-align: center;\n",
       "                    font-size: 16px;\n",
       "                    font-weight: bold;\n",
       "                    color: #555;\n",
       "                    margin: 10px 0;'>\n",
       "                    ... 295 more rows not displayed ...\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "35.0"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(optimized_react2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5) Zero-Shot Aggregator.\n",
    "\n",
    "Let's now extract the best five bootstrapped ReAct programs. We'll build a simple DSPy aggregator that runs all of them then produces a final answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dsp.utils import flatten, deduplicate\n",
    "\n",
    "# the best-performing five ReAct programs from the optimization process\n",
    "AGENTS = [x[-1] for x in optimized_react2.candidate_programs[:5]]\n",
    "\n",
    "class Aggregator(dspy.Module):\n",
    "\tdef __init__(self, temperature=0.0):\n",
    "\t\tself.aggregate = dspy.ChainOfThought('context, question -> answer')\n",
    "\t\tself.temperature = temperature\n",
    "\n",
    "\tdef forward(self, question):\n",
    "\t\t# Run all five agents with high temperature, then extract and deduplicate their observed contexts\n",
    "\t\twith dspy.context(lm=gpt3.copy(temperature=self.temperature)):\n",
    "\t\t\tpreds = [agent(question=question) for agent in AGENTS]\n",
    "\t\t\tcontext = deduplicate(flatten([flatten(p.observations) for p in preds]))\n",
    "\n",
    "\t\t# Run the aggregation step to produce a final answer\n",
    "\t\treturn self.aggregate(context=context, question=question)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's quickly evaluate the aggregator prior to optimization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 64 / 300  (21.3): 100%|██████████| 300/300 [12:53<00:00,  2.58s/it]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_07587 th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_07587 td {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_07587_row0_col0, #T_07587_row0_col1, #T_07587_row0_col2, #T_07587_row0_col3, #T_07587_row0_col4, #T_07587_row0_col5, #T_07587_row1_col0, #T_07587_row1_col1, #T_07587_row1_col2, #T_07587_row1_col3, #T_07587_row1_col4, #T_07587_row1_col5, #T_07587_row2_col0, #T_07587_row2_col1, #T_07587_row2_col2, #T_07587_row2_col3, #T_07587_row2_col4, #T_07587_row2_col5, #T_07587_row3_col0, #T_07587_row3_col1, #T_07587_row3_col2, #T_07587_row3_col3, #T_07587_row3_col4, #T_07587_row3_col5, #T_07587_row4_col0, #T_07587_row4_col1, #T_07587_row4_col2, #T_07587_row4_col3, #T_07587_row4_col4, #T_07587_row4_col5 {\n",
       "  text-align: left;\n",
       "  white-space: pre-wrap;\n",
       "  word-wrap: break-word;\n",
       "  max-width: 400px;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_07587\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_07587_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
       "      <th id=\"T_07587_level0_col1\" class=\"col_heading level0 col1\" >example_answer</th>\n",
       "      <th id=\"T_07587_level0_col2\" class=\"col_heading level0 col2\" >gold_titles</th>\n",
       "      <th id=\"T_07587_level0_col3\" class=\"col_heading level0 col3\" >rationale</th>\n",
       "      <th id=\"T_07587_level0_col4\" class=\"col_heading level0 col4\" >pred_answer</th>\n",
       "      <th id=\"T_07587_level0_col5\" class=\"col_heading level0 col5\" >answer_exact_match</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_07587_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_07587_row0_col0\" class=\"data row0 col0\" >Are both Cangzhou and Qionghai in the Hebei province of China?</td>\n",
       "      <td id=\"T_07587_row0_col1\" class=\"data row0 col1\" >no</td>\n",
       "      <td id=\"T_07587_row0_col2\" class=\"data row0 col2\" >{'Qionghai', 'Cangzhou'}</td>\n",
       "      <td id=\"T_07587_row0_col3\" class=\"data row0 col3\" >determine whether both Cangzhou and Qionghai are in the Hebei province of China. We know that Cangzhou is a prefecture-level city in eastern Hebei province,...</td>\n",
       "      <td id=\"T_07587_row0_col4\" class=\"data row0 col4\" >No, only Cangzhou is in Hebei province, while Qionghai is not.assistant</td>\n",
       "      <td id=\"T_07587_row0_col5\" class=\"data row0 col5\" >False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_07587_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_07587_row1_col0\" class=\"data row1 col0\" >Who conducts the draft in which Marc-Andre Fleury was drafted to the Vegas Golden Knights for the 2017-18 season?</td>\n",
       "      <td id=\"T_07587_row1_col1\" class=\"data row1 col1\" >National Hockey League</td>\n",
       "      <td id=\"T_07587_row1_col2\" class=\"data row1 col2\" >{'2017–18 Pittsburgh Penguins season', '2017 NHL Expansion Draft'}</td>\n",
       "      <td id=\"T_07587_row1_col3\" class=\"data row1 col3\" >answer this question. We know that Marc-Andre Fleury was drafted to the Vegas Golden Knights for the 2017-18 season. We also know that the Vegas...</td>\n",
       "      <td id=\"T_07587_row1_col4\" class=\"data row1 col4\" >The National Hockey League (NHL).assistant`</td>\n",
       "      <td id=\"T_07587_row1_col5\" class=\"data row1 col5\" >False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_07587_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_07587_row2_col0\" class=\"data row2 col0\" >The Wings entered a new era, following the retirement of which Canadian retired professional ice hockey player and current general manager of the Tampa Bay...</td>\n",
       "      <td id=\"T_07587_row2_col1\" class=\"data row2 col1\" >Steve Yzerman</td>\n",
       "      <td id=\"T_07587_row2_col2\" class=\"data row2 col2\" >{'2006–07 Detroit Red Wings season', 'Steve Yzerman'}</td>\n",
       "      <td id=\"T_07587_row2_col3\" class=\"data row2 col3\" >answer this question. We know that the Wings entered a new era following the retirement of a Canadian retired professional ice hockey player and current...</td>\n",
       "      <td id=\"T_07587_row2_col4\" class=\"data row2 col4\" >Steve Yzerman.</td>\n",
       "      <td id=\"T_07587_row2_col5\" class=\"data row2 col5\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_07587_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_07587_row3_col0\" class=\"data row3 col0\" >What river is near the Crichton Collegiate Church?</td>\n",
       "      <td id=\"T_07587_row3_col1\" class=\"data row3 col1\" >the River Tyne</td>\n",
       "      <td id=\"T_07587_row3_col2\" class=\"data row3 col2\" >{'Crichton Collegiate Church', 'Crichton Castle'}</td>\n",
       "      <td id=\"T_07587_row3_col3\" class=\"data row3 col3\" >find the answer. We know that the Crichton Collegiate Church is situated near the hamlet of Crichton in Midlothian, Scotland. We also know that the...</td>\n",
       "      <td id=\"T_07587_row3_col4\" class=\"data row3 col4\" >The River Esk.</td>\n",
       "      <td id=\"T_07587_row3_col5\" class=\"data row3 col5\" >False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_07587_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_07587_row4_col0\" class=\"data row4 col0\" >In the 10th Century A.D. Ealhswith had a son called Æthelweard by which English king?</td>\n",
       "      <td id=\"T_07587_row4_col1\" class=\"data row4 col1\" >King Alfred the Great</td>\n",
       "      <td id=\"T_07587_row4_col2\" class=\"data row4 col2\" >{'Æthelweard (son of Alfred)', 'Ealhswith'}</td>\n",
       "      <td id=\"T_07587_row4_col3\" class=\"data row4 col3\" >find the answer. We know that Ealhswith was the wife of King Alfred the Great. We also know that Æthelweard was the younger son of...</td>\n",
       "      <td id=\"T_07587_row4_col4\" class=\"data row4 col4\" >King Alfred the Great.</td>\n",
       "      <td id=\"T_07587_row4_col5\" class=\"data row4 col5\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7fa54bd7c070>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <div style='\n",
       "                    text-align: center;\n",
       "                    font-size: 16px;\n",
       "                    font-weight: bold;\n",
       "                    color: #555;\n",
       "                    margin: 10px 0;'>\n",
       "                    ... 295 more rows not displayed ...\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "21.33"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aggregator = Aggregator()\n",
    "evaluate(aggregator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6) Optimized Aggregator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 0 / 1  (0.0):   0%|          | 0/50 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 9 / 50  (18.0): 100%|██████████| 50/50 [01:24<00:00,  1.69s/it]\n",
      "Average Metric: 26 / 50  (52.0): 100%|██████████| 50/50 [00:15<00:00,  3.17it/s]\n",
      "  3%|▎         | 4/150 [01:18<48:03, 19.75s/it]\n",
      "Average Metric: 25 / 50  (50.0): 100%|██████████| 50/50 [00:09<00:00,  5.27it/s]\n",
      "  5%|▌         | 8/150 [02:15<40:06, 16.95s/it]\n",
      "Average Metric: 25 / 50  (50.0): 100%|██████████| 50/50 [00:16<00:00,  3.09it/s]\n",
      "  1%|          | 1/150 [00:18<47:10, 19.00s/it]\n",
      "Average Metric: 28 / 50  (56.0): 100%|██████████| 50/50 [00:12<00:00,  3.88it/s]\n",
      "  1%|          | 1/150 [00:01<03:32,  1.42s/it]\n",
      "Average Metric: 26 / 50  (52.0): 100%|██████████| 50/50 [00:12<00:00,  4.14it/s]\n",
      "  1%|          | 1/150 [00:16<40:39, 16.37s/it]\n",
      "Average Metric: 26 / 50  (52.0): 100%|██████████| 50/50 [00:15<00:00,  3.24it/s]\n",
      "  3%|▎         | 4/150 [01:52<1:08:19, 28.08s/it]\n",
      "Average Metric: 27 / 50  (54.0): 100%|██████████| 50/50 [00:17<00:00,  2.87it/s]\n",
      "  1%|▏         | 2/150 [00:29<35:48, 14.52s/it]\n",
      "Average Metric: 27 / 50  (54.0): 100%|██████████| 50/50 [00:14<00:00,  3.54it/s]\n",
      "  2%|▏         | 3/150 [01:10<57:42, 23.56s/it]  \n",
      "Average Metric: 25 / 50  (50.0): 100%|██████████| 50/50 [00:10<00:00,  4.65it/s]\n",
      "  3%|▎         | 4/150 [01:07<40:58, 16.84s/it]\n",
      "Average Metric: 27 / 50  (54.0): 100%|██████████| 50/50 [00:10<00:00,  4.64it/s]\n",
      "  1%|▏         | 2/150 [00:33<40:56, 16.60s/it]\n",
      "Average Metric: 28 / 50  (56.0): 100%|██████████| 50/50 [00:11<00:00,  4.42it/s]\n",
      "  2%|▏         | 3/150 [00:59<48:17, 19.71s/it]\n",
      "Average Metric: 26 / 50  (52.0): 100%|██████████| 50/50 [00:15<00:00,  3.30it/s]\n"
     ]
    }
   ],
   "source": [
    "kwargs = dict(max_bootstrapped_demos=2, max_labeled_demos=6, num_candidate_programs=10, num_threads=8)\n",
    "tp = BootstrapFewShotWithRandomSearch(metric=dspy.evaluate.answer_exact_match, **kwargs)\n",
    "optimized_aggregator = tp.compile(aggregator, trainset=trainset, valset=valset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/300 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 176 / 300  (58.7): 100%|██████████| 300/300 [11:59<00:00,  2.40s/it]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_c67a9 th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_c67a9 td {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_c67a9_row0_col0, #T_c67a9_row0_col1, #T_c67a9_row0_col2, #T_c67a9_row0_col3, #T_c67a9_row0_col4, #T_c67a9_row0_col5, #T_c67a9_row1_col0, #T_c67a9_row1_col1, #T_c67a9_row1_col2, #T_c67a9_row1_col3, #T_c67a9_row1_col4, #T_c67a9_row1_col5, #T_c67a9_row2_col0, #T_c67a9_row2_col1, #T_c67a9_row2_col2, #T_c67a9_row2_col3, #T_c67a9_row2_col4, #T_c67a9_row2_col5, #T_c67a9_row3_col0, #T_c67a9_row3_col1, #T_c67a9_row3_col2, #T_c67a9_row3_col3, #T_c67a9_row3_col4, #T_c67a9_row3_col5, #T_c67a9_row4_col0, #T_c67a9_row4_col1, #T_c67a9_row4_col2, #T_c67a9_row4_col3, #T_c67a9_row4_col4, #T_c67a9_row4_col5 {\n",
       "  text-align: left;\n",
       "  white-space: pre-wrap;\n",
       "  word-wrap: break-word;\n",
       "  max-width: 400px;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_c67a9\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_c67a9_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
       "      <th id=\"T_c67a9_level0_col1\" class=\"col_heading level0 col1\" >example_answer</th>\n",
       "      <th id=\"T_c67a9_level0_col2\" class=\"col_heading level0 col2\" >gold_titles</th>\n",
       "      <th id=\"T_c67a9_level0_col3\" class=\"col_heading level0 col3\" >rationale</th>\n",
       "      <th id=\"T_c67a9_level0_col4\" class=\"col_heading level0 col4\" >pred_answer</th>\n",
       "      <th id=\"T_c67a9_level0_col5\" class=\"col_heading level0 col5\" >answer_exact_match</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_c67a9_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_c67a9_row0_col0\" class=\"data row0 col0\" >Are both Cangzhou and Qionghai in the Hebei province of China?</td>\n",
       "      <td id=\"T_c67a9_row0_col1\" class=\"data row0 col1\" >no</td>\n",
       "      <td id=\"T_c67a9_row0_col2\" class=\"data row0 col2\" >{'Qionghai', 'Cangzhou'}</td>\n",
       "      <td id=\"T_c67a9_row0_col3\" class=\"data row0 col3\" >find the answer. We know that Cangzhou is a prefecture-level city in eastern Hebei province, while Qionghai is one of the seven county-level cities of...</td>\n",
       "      <td id=\"T_c67a9_row0_col4\" class=\"data row0 col4\" >No</td>\n",
       "      <td id=\"T_c67a9_row0_col5\" class=\"data row0 col5\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_c67a9_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_c67a9_row1_col0\" class=\"data row1 col0\" >Who conducts the draft in which Marc-Andre Fleury was drafted to the Vegas Golden Knights for the 2017-18 season?</td>\n",
       "      <td id=\"T_c67a9_row1_col1\" class=\"data row1 col1\" >National Hockey League</td>\n",
       "      <td id=\"T_c67a9_row1_col2\" class=\"data row1 col2\" >{'2017–18 Pittsburgh Penguins season', '2017 NHL Expansion Draft'}</td>\n",
       "      <td id=\"T_c67a9_row1_col3\" class=\"data row1 col3\" >find the answer. We know that Marc-André Fleury was drafted to the Vegas Golden Knights in the 2017 NHL Expansion Draft, and we also know...</td>\n",
       "      <td id=\"T_c67a9_row1_col4\" class=\"data row1 col4\" >The National Hockey League</td>\n",
       "      <td id=\"T_c67a9_row1_col5\" class=\"data row1 col5\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_c67a9_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_c67a9_row2_col0\" class=\"data row2 col0\" >The Wings entered a new era, following the retirement of which Canadian retired professional ice hockey player and current general manager of the Tampa Bay...</td>\n",
       "      <td id=\"T_c67a9_row2_col1\" class=\"data row2 col1\" >Steve Yzerman</td>\n",
       "      <td id=\"T_c67a9_row2_col2\" class=\"data row2 col2\" >{'2006–07 Detroit Red Wings season', 'Steve Yzerman'}</td>\n",
       "      <td id=\"T_c67a9_row2_col3\" class=\"data row2 col3\" >find the answer. We know that the Wings entered a new era, following the retirement of Steve Yzerman, and we also know that Yzerman is...</td>\n",
       "      <td id=\"T_c67a9_row2_col4\" class=\"data row2 col4\" >Steve Yzerman</td>\n",
       "      <td id=\"T_c67a9_row2_col5\" class=\"data row2 col5\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_c67a9_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_c67a9_row3_col0\" class=\"data row3 col0\" >What river is near the Crichton Collegiate Church?</td>\n",
       "      <td id=\"T_c67a9_row3_col1\" class=\"data row3 col1\" >the River Tyne</td>\n",
       "      <td id=\"T_c67a9_row3_col2\" class=\"data row3 col2\" >{'Crichton Collegiate Church', 'Crichton Castle'}</td>\n",
       "      <td id=\"T_c67a9_row3_col3\" class=\"data row3 col3\" >find the answer. We know that the Crichton Collegiate Church is situated in Midlothian, Scotland, and we also know that the River Esk flows through...</td>\n",
       "      <td id=\"T_c67a9_row3_col4\" class=\"data row3 col4\" >the River Esk</td>\n",
       "      <td id=\"T_c67a9_row3_col5\" class=\"data row3 col5\" >False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_c67a9_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_c67a9_row4_col0\" class=\"data row4 col0\" >In the 10th Century A.D. Ealhswith had a son called Æthelweard by which English king?</td>\n",
       "      <td id=\"T_c67a9_row4_col1\" class=\"data row4 col1\" >King Alfred the Great</td>\n",
       "      <td id=\"T_c67a9_row4_col2\" class=\"data row4 col2\" >{'Æthelweard (son of Alfred)', 'Ealhswith'}</td>\n",
       "      <td id=\"T_c67a9_row4_col3\" class=\"data row4 col3\" >find the answer. We know that Ealhswith was the wife of King Alfred the Great, and we also know that Æthelweard was the younger son...</td>\n",
       "      <td id=\"T_c67a9_row4_col4\" class=\"data row4 col4\" >King Alfred the Great</td>\n",
       "      <td id=\"T_c67a9_row4_col5\" class=\"data row4 col5\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7fa56c2ae050>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <div style='\n",
       "                    text-align: center;\n",
       "                    font-size: 16px;\n",
       "                    font-weight: bold;\n",
       "                    color: #555;\n",
       "                    margin: 10px 0;'>\n",
       "                    ... 295 more rows not displayed ...\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "58.67"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "optimized_aggregator2 = optimized_aggregator.deepcopy()\n",
    "optimized_aggregator2.temperature = 0.7\n",
    "\n",
    "evaluate(optimized_aggregator2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7) Conclusion.\n",
    "\n",
    "Normally, we like to release notebooks with pre-computed caches and to inspect the prompts with `llama3.inspect_history` to explore the behavior of optimization. See the intro notebook (or any of the Colab notebooks on the README) for such annotated examples!\n",
    "\n",
    "To keep the current release super quick, Omar will extend this notebook into an annotated version if there's significant interest."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py310",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
